-
Notifications
You must be signed in to change notification settings - Fork 9
Initial multicast support #847
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
FelixMcFelix
wants to merge
8
commits into
master
Choose a base branch
from
mcast
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
+6,461
−467
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also pushes on the requisite extensions for us to fill in
This implements IPv4 and IPv6 multicast packet forwarding with three
replication modes (External, Underlay, All) for rack-wide multicast
delivery across VPCs.
Includes:
- M2P (Multicast-to-Physical) mappings with admin-scoped IPv6 underlay
- Per-port multicast group subscriptions for local delivery
- Multicast forwarding table with configurable replication strategies
- Geneve multicast option encoding for delivery mode signaling
- RX path loop prevention (packets marked Underlay skip re-relay)
- TX/RX path integration with flow table and encapsulation
- DTrace probes for multicast delivery observability
- API addition: set_mcast_fwd/clear_mcast_fwd for forwarding table management
- API addition: mcast_subscribe/mcast_unsubscribe for port group membership
- API addition: dump_mcast_fwd for observability
- Testing: XDE integration tests covering all replication modes, validation,
and edge cases
- Testing: oxide-vpc integration tests for Geneve encapsulation and parsing
- Enforce DEFAULT_MULTICAST_VNI (77) for all multicast traffic (groups
are fleet-side/cross-VPC) and validate admin-scoped underlay
addresses (ff04::/16, ff05::/16, ff08::/16).
FelixMcFelix
commented
Oct 21, 2025
Collaborator
Author
FelixMcFelix
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the work so far. I haven't looked at the new multicast integration tests yet, but I have a bunch of questions from what I've looked at thus far.
Updates all-around for IPv4/IPv6 multicast support with control-plane APIs,
kernel TX/RX implementation, dtrace script, and documentation semantics.
Includes:
- Delivery semantics (leaf-node):
- Remove multicast relay logic; OPTE is always a leaf node in the
replication tree
- Same-sled delivery happens unconditionally on TX for local subscribers
- RX-path only handles packets destined to this sled (no forwarding)
- Perf (avoid management_lock in datapath):
- Move mcast_fwd lookups to per-entry state instead of hitting
exclusive management lock during TX replication
- Clone Arc references from per-CPU caches instead of holding per-port
RwLock guards across packet processing
- Use state.devs.read() for concurrent dataplane access
- Hold per-CPU copies of mcast_fwd for duration of TX replication
- Arbitrary VNI handling:
- Use DEFAULT_MULTICAST_VNI (77) for fleet-wide multicast delivery
- Remove per-VPC VNI checks in xde.rs; delegate validation to overlay layer
- Packets with VNI 77 delivered to all subscribers regardless of VPC
- Replication flag clarification:
- Replication enum specifies switch behavior on marked packets:
- External: Switch replicates to front panel ports (leaving underlay)
- Underlay: Switch replicates to sleds (within underlay)
- Both: Switch does both replications
- Used only on TX-path to inform switch behavior, not for RX-path
- Routing and MACs:
- Now, we set the right nexthop and routing for TX replication (the
switch unicast address)
- Use derived IPv6 multicast MAC for outer destination
- Route lookup determines underlay port selection via next_hop
- Simplified underlay routing for admin-scoped (ff04::/16)
addresses, matching Omicron currently
- Test infra:
- MulticastGroup: RAII cleanup for M2P/forwarding entries
- SnoopGuard: Prevent leaked snoop processes from holding DLPI devices
- Geneve packet verification with replication flag validation
- three_node_topology for multi-subscriber scenarios
- Proactive zone cleanup
- Standardized around updated semantics
- Additional refinements:
- Updated DTrace script (opte-mcast-delivery.d)
- Improved opteadm output formatting for multicast commands
- Added anyhow dependency to opte-test-utils
- Updated documentation clarifying multicast architecture
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
XXX currently sketching out the bare minimum requirements around frame delivery, insertion of Geneve options, etc.
TODO:
Ipv6Addr->BTreeMap<(NextHopV6, Replication)>).Mcast2Physand above table via ioctl.